A nuanced view of the extent to which samples from narrow populations are scientifically problematic.

Psychologists have a traditional concern with participant samples from narrow populations and deleterious effects on researchers' ability to generalize findings. Recently, both individuals and authoritative organizations, such as the American Psychological Association, have merged this external validity concern with diversity and inclusion concerns. The American Psychological Association directive for researchers to include diverse samples seems obviously well-taken as it purports to mitigate these problems at once; it simultaneously increases external validity and promotes diversity and inclusion. However, we show that there are complications. These include problems with internal and external validity conceptualizations; that sometimes generalization failures can support, rather than detract from, external validity; the crucial role auxiliary assumptions play in impacting internal and external validity; Lakatosian degenerative science and its problematic application; and distinguishing between merely including diverse groups in research samples versus analyzing for group differences. These complications imply a nuanced perspective of whether samples from narrow populations are undesirable. That a sample is from a narrow population might, or might not, preclude strong support or disconfirmation for the theory, including its ability to generalize. Our nuanced perspective militates against the current trend of journal directives to require diverse samples. Sample suitability for particular researcher goals should be judged on a case-by-case basis that takes into account that sometimes samples from narrow populations can nevertheless engender impressive scientific progress and sometimes not. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The role of free will beliefs in social behavior: Priority areas for future research.

Recent research has examined the consequences that holding views about free will has on social behavior. Specifically, through manipulating people's belief in free will, researchers have tested the psychological and behavioral consequences of free will belief change. However, findings of such manipulations have been shown to be relatively small and inconsistent. The purpose of this paper is to outline four key areas for researchers in this area to consider. We believe considering these areas will give a more nuanced understanding of the role of free will beliefs.

Enhancing predictive power by unamalgamating multi-item scales.

The generally small but touted as "statistically significant" correlation coefficients in the social sciences jeopardize theory testing and prediction. To investigate these small coefficients' underlying causes, traditional equations such as Spearman's (1904) classic attenuation formula, Cronbach's (1951) alpha, and Guilford and Fruchter's (1973) equation for the effect of additional items on a scale's predictive power are considered. These equations' implications differ regarding large interitem correlations enhancing or diminishing predictive power. Contrary to conventional practice, such correlations decrease predictive power when treating items as multi-item scale components but can increase predictive power when treating items separately. The implications are wide-ranging. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A New Way to Think About Internal and External Validity.

Researchers have been concerned with internal and external validity for decades, and the discussion continues. The present proposal is that there are less important and more important senses in which one can interpret internal and external validity, and these can be integrated with a taxonomy that includes theoretical, auxiliary, statistical, and inferential assumptions. The integration sheds light on recent exchanges in the literature on validity and suggests that the vaunted internal-external validity trade-off is false for more important senses of internal and external validity; internal and external validity increase or decrease together when there is an emphasis on underlying theories. Finally, the integration implies the desirability of some changes in typical research advice and practice.

Why we habitually engage in null-hypothesis significance testing: A qualitative study.

BACKGROUND: Null Hypothesis Significance Testing (NHST) is the most familiar statistical procedure for making inferences about population effects. Important problems associated with this method have been addressed and various alternatives that overcome these problems have been developed. Despite its many well-documented drawbacks, NHST remains the prevailing method for drawing conclusions from data. Reasons for this have been insufficiently investigated. Therefore, the aim of our study was to explore the perceived barriers and facilitators related to the use of NHST and alternative statistical procedures among relevant stakeholders in the scientific system. METHODS: Individual semi-structured interviews and focus groups were conducted with junior and senior researchers, lecturers in statistics, editors of scientific journals and program leaders of funding agencies. During the focus groups, important themes that emerged from the interviews were discussed. Data analysis was performed using the constant comparison method, allowing emerging (sub)themes to be fully explored. A theory substantiating the prevailing use of NHST was developed based on the main themes and subthemes we identified. RESULTS: Twenty-nine interviews and six focus groups were conducted. Several interrelated facilitators and barriers associated with the use of NHST and alternative statistical procedures were identified. These factors were subsumed under three main themes: the scientific climate, scientific duty, and reactivity. As a result of the factors, most participants feel dependent in their actions upon others, have become reactive, and await action and initiatives from others. This may explain why NHST is still the standard and ubiquitously used by almost everyone involved. CONCLUSION: Our findings demonstrate how perceived barriers to shift away from NHST set a high threshold for actual behavioral change and create a circle of interdependency between stakeholders. By taking small steps it should be possible to decrease the scientific community's strong dependence on NHST and p-values.

Making the A Priori Procedure Work for Differences Between Means.

Previous researchers have proposed the a priori procedure, whereby the researcher specifies, prior to data collection, how closely she wishes the sample means to approach corresponding population means, and the degree of confidence of meeting the specification. However, an important limitation of previous research is that researchers sometimes are interested in differences between means, rather than in the means themselves. To address this limitation, we propose additional equations that expand the a priori procedure to handle differences between means, both in matched and in independent samples. Finally, implications are discussed.

From a Sampling Precision Perspective, Skewness Is a Friend and Not an Enemy!

Two recent publications in Educational and Psychological Measurement advocated that researchers consider using the a priori procedure. According to this procedure, the researcher specifies, prior to data collection, how close she wishes her sample mean(s) to be to the corresponding population mean(s), and the desired probability of being that close. A priori equations provide the necessary sample size to meet specifications under the normal distribution. Or, if sample size is taken as given, a priori equations provide the precision with which estimates of distribution means can be made. However, there is currently no way to perform these calculations under the more general family of skew-normal distributions. The present research provides the necessary equations. In addition, we show how skewness can increase the precision with which locations of distributions can be estimated. This conclusion, based on the perspective of improving sampling precision, contrasts with a typical argument in favor of performing transformations to normalize skewed data for the sake of performing more efficient significance tests.

The sampling precision of research in five major areas of psychology.

After obtaining a sample of published, peer-reviewed articles from journals with high and low impact factors in social, cognitive, neuro-, developmental, and clinical psychology, we used a priori equations recently derived by Trafimow (Educational and Psychological Measurement, 77, 831-854, 2017; Trafimow & MacDonald in Educational and Psychological Measurement, 77, 204-219, 2017) to compute the articles' median levels of precision. Our findings indicate that developmental research performs best with respect to precision, whereas cognitive research performs the worst; however, none of the psychology subfields excelled. In addition, we found important differences in precision between journals in the upper versus lower echelons with respect to impact factors in cognitive, neuro-, and clinical psychology, whereas the difference was dramatically attenuated for social and developmental psychology. Implications are discussed.

Some Implications of Distinguishing Between Unexplained Variance That Is Systematic or Random.

Because error variance alternatively can be considered to be the sum of systematic variance associated with unknown variables and randomness, a tripartite assumption is proposed that total variance in the dependent variable can be partitioned into three variance components. These are variance in the dependent variable that is explained by the independent variable, variance in the dependent variable that is unexplained but systematic (associated with variance in unknown variables), and random variance. Based on the tripartite assumption, classical measurement theory, and simple mathematics, it is shown that these components can be estimated using observable data. Mathematical and computer simulations illustrate some of the important issues and implications.

Manipulating the Alpha Level Cannot Cure Significance Testing.

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable.

A close consideration of effect sizes reviewed by Jussim (2012).

This commentary on Jussim (2012) makes two points: (1) Effect sizes often reflect artifacts of experimental design rather than real-world relevance, and (2) any argument dependent on effect sizes must correct for attenuation due to instrument reliabilities. A formula for making this correction is presented, and its ramifications on the debate over accuracy in person perception are discussed.

The probability of simple versus complex causal models in causal analyses.

Complex causal models, accompanied by causal analyses based on large correlation matrices, are more common in the social sciences than are simple causal models accompanied by a single correlation coefficient. The increased complexity of the former, relative to the latter, seems to carry with it an augmented scientific respectability or credibility. In contrast, the axioms of probability suggest an argument in the opposite direction.

Performing Inferential Statistics Prior to Data Collection.

Typically, in education and psychology research, the investigator collects data and subsequently performs descriptive and inferential statistics. For example, a researcher might compute group means and use the null hypothesis significance testing procedure to draw conclusions about the populations from which the groups were drawn. We propose an alternative inferential statistical procedure that is performed prior to data collection rather than afterwards. To use this procedure, the researcher specifies how close she or he desires the group means to be to their corresponding population means and how confident she or he wishes to be that this actually is so. We derive an equation that provides researchers with a way to determine the sample size needed to meet the specifications concerning closeness and confidence, regardless of the number of groups.

Using the Coefficient of Confidence to Make the Philosophical Switch From A Posteriori to A Priori Inferential Statistics.

There has been much controversy over the null hypothesis significance testing procedure, with much of the criticism centered on the problem of inverse inference. Specifically, p gives the probability of the finding (or one more extreme) given the null hypothesis, whereas the null hypothesis significance testing procedure involves drawing a conclusion about the null hypothesis given the finding. Many critics have called for null hypothesis significance tests to be replaced with confidence intervals. However, confidence intervals also suffer from a version of the inverse inference problem. The only known solution to the inverse inference problem is to use the famous theorem by Bayes, but this involves commitments that many researchers are not willing to make. However, it is possible to ask a useful question for which inverse inference is not a problem and that leads to the computation of the coefficient of confidence. In turn, and much more important, using the coefficient of confidence implies the desirability of switching from the current emphasis on a posteriori inferential statistics to an emphasis on a priori inferential statistics.

Evolutionary Influences on Attribution and Affect.

Evolutionary theory was applied to Reeder and Brewer's schematic theory and Trafimow's affect theory to extend this area of research with five new predictions involving affect and ability attributions, comparing morality and ability attributions, gender differences, and reaction times for affect and attribution ratings. The design included a 2 (Trait Dimension Type: HR, PR) × 2 (Behavior Type: morality, ability) × 2 (Valence: positive, negative) × 2 (Replication: original, replication) × 2 (Sex: female or male actor) × 2 (Gender: female or male participant) × 2 (Order: attribution portion first, affect portion first) mixed design. All factors were within participants except the order and participant gender. Participants were presented with 32 different scenarios in which an actor engaged in a concrete behavior after which they made attributions and rated their affect in response to the behavior. Reaction times were measured during attribution and affect ratings. In general, the findings from the experiment supported the new predictions. Affect was related to attributions for both morality and ability related behaviors. Morality related behaviors received more extreme attribution and affect ratings than ability related behaviors. Female actors received stronger attribution and affect ratings for diagnostic morality behaviors compared to male actors. Male and female actors received similar attribution and affect ratings for diagnostic ability behaviors. Diagnostic behaviors were associated with lower reaction times than non-diagnostic behaviors. These findings demonstrate the utility of evolutionary theory in creating new hypotheses and empirical findings in the domain of attribution.